Papers with memory usage
Computationally Efficient Wasserstein Loss for Structured Labels (2021.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to estimate the probability distribution of labels are based on tree-Wasserstein distance. |
| Approach: | They propose a tree-Wasserstein distance regularized LDL algorithm for hierarchical text classification tasks. |
| Outcome: | The proposed method performs well on synthetic and real-world datasets and compares favorably with the Sinkhorn algorithm in terms of computation time and memory usage. |
Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality (P18-2)
Copied to clipboard
| Challenge: | Recurrent neural tensor networks (RNNs) increase capacity by augmenting the size of the hidden layer, with significant increase in computational cost. |
| Approach: | They propose restricted recurrent neural tensor networks (r-RNTNs) which reserve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights . |
| Outcome: | The proposed model outperforms unrestricted RNTNs using only a small fraction of the parameters of unrestrained RNNNs. |
Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021.eacl-demos)
Copied to clipboard
| Challenge: | Trankit is a lightweight, pre-trained toolkit for multilingual natural language processing. |
| Approach: | They propose a transformer-based toolkit for multilingual natural language processing that trains pipelines over 100 languages and 90 pretrained pipelines for 56 languages. |
| Outcome: | The proposed tool outperforms existing pipelines over sentence segmentation, part-of-speech tagging, morphological feature tabbing, and dependency parsing while maintaining competitive performance over tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. |
AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end speech translation (ST) models need large amount of training data to perform well. |
| Approach: | They propose a shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. |
| Outcome: | The proposed method achieves better performance on the MUST-C dataset, with higher inference speed and lower memory usage. |
Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | AMR-to-text generation is used to transduce Abstract Meaning Representation structures (AMRs) Graph Convolution Networks (GCNs) are not able to capture non-local information and follow a local (first-order) information aggregation scheme. |
| Approach: | They propose a dynamic fusion mechanism that captures richer non-local interactions . they propose weight tied convolutions and group graph convolution to reduce memory usage . |
| Outcome: | The proposed model outperforms state-of-the-art models on two benchmark datasets with significantly fewer parameters while maintaining the model capacity. |
Beyond Dynamic Quantization: An Efficient Static Hierarchical Mix-precision Framework for Near-Lossless LLM Compression (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for dynamic quantization are hardware-unfriendly and often lead to large quantization errors in static scenarios. |
| Approach: | They propose a Static Hierarchical Mix-precision Quantization method which quantifies both inter-layer and intra-layer sensitivity through unified derivations involving Hessian. |
| Outcome: | The proposed method achieves 75.58% on zero-shot reasoning tasks while yielding average speedup of 2.86. |
UNComp: Can Matrix Entropy Uncover Sparsity? — A Compressor Design from an Uncertainty-Aware Perspective (2025.emnlp-main)
Copied to clipboard
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, Lingpeng Kong, Ngai Wong
| Challenge: | Deploying large language models (LLMs) for long-context inference remains challenging due to their substantial memory and computational demands. |
| Approach: | They propose an uncertainty-aware framework that leverages truncated matrix entropy to identify areas of low information content. |
| Outcome: | The proposed framework reduces the KV cache size to 4.74% of the original and achieves a 6% speedup. |
Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models overlook intrinsic semantic associations between soft prompt tokens, leading to high discreteness and limited interactions. |
| Approach: | They propose a low-parameters Prompt Tuning method which leverages prompt decomposition and compressed outer product to facilitate multiple interactions among prompt tokens. |
| Outcome: | Experiments on six architectures and eight datasets show that the proposed method outperforms state-of-the-art methods in performance and efficiency. |
Teacher and Student Models of Offensive Language in Social Media (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to identify offensive language online use large pre-trained transformer models. however, the inference time, disk, and memory requirements of these models are prohibitively large. |
| Approach: | They propose to transfer knowledge from large transformer models to much smaller neural models to make predictions at the token- and post-level. |
| Outcome: | The proposed model performs 100 times better than transformer models but with 100 times less parameters and much less memory usage. |
Cost-Optimal Grouped-Query Attention for Long-Context Modeling (2025.emnlp-main)
Copied to clipboard
Yingfa Chen, Yutong Wu, Chenyang Song, Zhen Leng Thai, Xingyu Shen, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Current GQA configurations overlook how context length influences inference cost . |
| Approach: | They propose a recipe for deriving cost-optimal GQA configurations that decouple the total head size from the hidden size and allow more flexible control over attention FLOPs. |
| Outcome: | The proposed configurations reduce memory usage and FLOPs by more than 50% compared to Llama-3's GQA, with *no degradation in model capabilities*. |
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding (2025.findings-naacl)
Copied to clipboard
| Challenge: | Multi-Layer Key-Value (MLKV) sharing reduces memory usage by 6x compared to Multi-Query Attention and Grouped-Query Attributes. |
| Approach: | They propose a novel approach that extends KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention and Grouped-Query Attributes. |
| Outcome: | The proposed approach reduces KV cache size by 6x with minimal performance loss and scales linearly with model size, batch size, and sequence length. |
Prompt Compression for Large Language Models: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input. |
| Approach: | They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs. |
| Outcome: | The proposed methods are categorized into hard prompt methods and soft prompt methods. |
MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight Quantization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for efficient deployment of small language models face inefficient bit-width allocation and insufficient fine-grained quantization adjustments. |
| Approach: | They propose a weight quantization technique that facilitates efficient deployment of SLMs . they propose to combine inter-layer loss and intra-layer salience to achieve better allocation . |
| Outcome: | Experimental results show that multi-level weight quantization achieves competitive performance compared to state-of-the-art methods. |
MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to optimize large language models for long-context inference are inefficient and consume memory. |
| Approach: | They propose a mixed-precision quantization method via mixture of experts that inputs tokens into router chunk by chunk to reduce inference overhead. |
| Outcome: | The proposed method outperforms state-of-the-art KV cache quantization methods on multiple benchmark datasets. |
Learning from Negative Samples in Biomedical Generative Entity Linking (2025.findings-acl)
Copied to clipboard
| Challenge: | Generative models are usually trained only with positive samples and do not explicitly learn from hard negative samples, which are entities that look similar but have different meanings. |
| Approach: | They propose a framework that trains generative BioEL models using negative samples to learn from hard negative samples. |
| Outcome: | The proposed framework outperforms baseline models by up to an average top-1 accuracy of 1.4% on five benchmarks. |
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation (2026.findings-acl)
Copied to clipboard
Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi, Hejian Sang, Shao Tang, Qingquan Song, Zhipeng Wang, Muhammad Abdul-Mageed
| Challenge: | a large reasoning model (LRM) training on large amounts of reasoning data is computationally expensive. |
| Approach: | They propose a method to quantify computation-quality tradeoffs as a function of sequence length. |
| Outcome: | The proposed method reduces training time, memory and FLOPs by 50% on long training sequences while retaining the full-sequence performance. |
VIP5: Towards Multimodal Foundation Models for Recommendation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in foundation models have impeded the ability for these fields to benefit from each other’s advancements. |
| Approach: | They propose to use a multimodal foundation model to unify various modalities and recommendation tasks under the P5 recommendation paradigm to implement personalized prompts. |
| Outcome: | The proposed model will unify visual, textual, and personalization modalities under the P5 recommendation paradigm and will improve recommendation performance and efficiency. |
LightThinker: Thinking Step-by-Step Compression (2025.emnlp-main)
Copied to clipboard
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in Large Language Models have demonstrated their remarkable capabilities in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. |
| Approach: | They propose a method that dynamically compresses verbose thought steps into compact representations and discards original reasoning chains. |
| Outcome: | The proposed method reduces peak memory usage and inference time while maintaining competitive accuracy. |
Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidence (2026.acl-long)
Copied to clipboard
| Challenge: | Guided by Gut (GG) is an efficient self-guided TTS framework for Large Language Models (LLMs) that performs step-by-step reasoning at a low cost without any reward models or verifiers. |
| Approach: | They propose a self-guided TTS framework that enables LLMs to perform step-by-step reasoning at a low cost without any reward models or verifiers. |
| Outcome: | Empirical evaluations show that GG performs better than TTS with PRMs while reducing GPU memory usage by up to 10. |
FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank compression can reduce memory usage and computational demand, but results are poor during decoding. |
| Approach: | They propose a fine-grained low-rank compression algorithm that determines optimal rank allocation for each layer and incorporates progressive low-ranked decoding to maintain text generation quality. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on summarization tasks and on understanding tasks. |
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Mixture-of-Experts (SMoE) architectures require loading all expert parameters . previous work focused on expert pruning and merging but focused on neuron-level structure . |
| Approach: | They propose a task-agnostic framework for expert pruning and reconstruction . it prunes redundant experts using router statistics, then decomposes them into neuron-level expert segments . |
| Outcome: | The proposed framework reduces the number of experts and memory usage, making it easier to deploy. |
GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing projection-based methods that project gradients into a lower-dimensional subspace can introduce computational and memory overheads. |
| Approach: | They propose a novel approach that leverages sparse projections to transform gradients into structured sparser updates. |
| Outcome: | The proposed approach significantly reduces memory usage for optimizer states and minimizes memory footprint, computation, and communication costs, leading to substantial throughput improvements. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass. |
| Approach: | They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens. |
| Outcome: | The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost . |
Value Residual Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing decoder-only transformers fail to preserve initial token-level information in deeper layers. |
| Approach: | They propose a new architecture that incorporates value residual connections in addition to hidden state residuals. |
| Outcome: | The proposed architecture reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods. |
FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel at handling long-context sequences, but require substantial prefill computation and key-value (KV) cache. |
| Approach: | They propose a KV cache compression framework that decouples prefill computation from decoding KV budget. |
| Outcome: | The proposed framework reduces latency in prefill and decoding by leveraging the stabilization of token importance in later layers. |
AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Weight-only quantization is a powerful optimization technique for large language models . pushing below 4 bits often leads to substantial accuracy degradation due to increased quantization error. |
| Approach: | They propose a framework that assigns layer-wise quantization bit-widths to optimize model quality and memory usage. |
| Outcome: | The proposed framework can optimize for large language models under memory constraints. |